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ABSTRACT 



This document reports on the conduct and findings of a 
3-year evaluation of the 1995-96 school year of the Learn and Serve America 
School and Community- Based Programs, which were initiated to support school 
and community-based efforts to involve school-aged youth in community 
service. Well-designed, school-based service-learning programs have a 
positive impact on young peoples' civic and educational attitudes and school 
performance while also meeting important community needs. While the impacts 
on participants tend to fade in the year after initial participation, 
students who continue their involvement in organized service do continue to 
show positive effects over the longer-term. When these programs are 
well-designed and full - implemented, they are more likely to produce impacts 
on participating youth, demonstrating the importance of continuing the 
emphasis on improving the quality of these programs to meet the goals of the 
national community service legislation. Thirty-seven tables and figures are 
included. The appendixes contain: site selection sampling strategy; impact 
estimation methods; definitions of scales and other outcome measures; 54 
tables detailing postprogram participant impacts, postprogram subgroup 
impacts, follow-up participant and subgroup effects, and follow-up effects 
for repeaters and nonrepeaters; and 4 survey instruments.) (MN) 
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Executive Summary 



Final Report 

Learn and Serve America 
School and Community-Based Programs 

In 1993, the National and Community Service Trust Act established the Learn and Serve 
America School and Community-Based Programs to support school and community-based efforts to 
involve school-aged youth in community service. The program is administered by the Corporation 
for National Service and funded through grants to states and national organizations, and through them 
to individual school districts, schools, and community organizations. In 1994-95, the first year of the 
program, the Corporation awarded approximately $30 million in grants supporting over 2,000 local 
efforts involving over 750,000 school-aged youth. 

Between 1994 and 1997, Brandeis University’s Center for Human Resources and Abt 
Associates Inc. conducted an evaluation of the national Learn and Serve America program for the 
Corporation for National Service. The evaluation was designed to address four fundamental 
questions: 

2. What is the impact of program participation on program participants? 

2. What are the institutional impacts on participating schools and community 

organizations? 

3. What impacts do Learn and Serve programs have on their communities? 

4. What is the return (in dollar terms) on the Learn and Serve investment? 

To answer these questions, the evaluation examined programs in seventeen middle and high 
school sites across the country using a variety of quantitative and qualitative methods. These included 
analysis of pre- and post-program surveys and school record data for approximately 1,000 Learn and 
Serve program participants and comparison group members; analysis of survey and school record data 
on approximately 760 participants and comparison group members at a one-year follow-up; analysis 
of teacher and community agency surveys from the seventeen sites; and on-site interviews and 
observation. The major focus for the evaluation was the 1995-96 school year, with student and 
teacher follow-up surveys taking place in Spring 1997. 

The purpose of this report is to present the findings from the evaluation. It provides 
information on four major areas of impact: short- and longer-term participant impacts, services 
provided to communities, impacts on participating schools, and an analysis of program return on 
investment. 

It is important to note that, in contrast to many national evaluations, this study does not focus 
on a representative sample of Learn and Serve programs. Instead, the evaluation focuses on a 
specific subset of "well-designed," or "high quality" programs. All of the programs selected for the 
study had been in operation for more than one year when selected and reported higher than average 
service hours and regular use of written and oral reflection. All were school-based initiatives and 
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linked to a formal course curriculum. As such, this evaluation is not intended to address the average 
impact of all Learn and Serve programs, but rather to identify the impacts that can be reasonably 
expected from mature, fully-implemented, school-based service-learning efforts. 

The major findings from the evaluation are as follows: 



Post-Program Participant Impacts 

The evaluation examined participants impacts at two points in time: immediately after 
program participation (short-term or post-program impacts) and one year after initial participation 
(impacts at follow-up). 

Based on the data from the 1995-96 school year, the Learn and Serve programs in this study 
had a positive post-program impact on the civic attitudes and educational development of program 
participants. At the end of the 1995-96 program year: 

• Learn and Serve participants in the study showed positive, statistically significant 
impacts on three of the four measures of civic attitudes used in the study: acceptance 
of cultural diversity, service leadership (defined as the degree to which students feel 
they are aware of needs in a community, are able to develop and implement a service 
project, and are committed to service now and later in life), and "civic attitudes," a 
measure that combines measures of service leadership, acceptance of diversity, and 
personal and social responsibility. 

• The Learn and Serve programs also involved participants in significantly more 
volunteer service than comparison group members. Participants were 20% more 
likely than comparison group members to have been involved in some form of 
volunteer service during the previous six months and provided more than twice as 
many hours of service during that time period. The data on hours show that service 
programs were not simply diverting students from other volunteer opportunities. 
Rather, they were increasing the number of students involved in service and 
significantly increasing the hours of service they provided. 

• Learn and Serve programs also had a positive effect on participants’ educational 
attitudes and school performance during program participation, with positive, 
statistically significant impacts on two measures — school engagement and math 
grades — and marginally significant impacts on science grades and core grade point 
average (English, math, science, and social studies grades combined). There were no 
significant impacts on English and social studies grades or measures of course failure, 
homework hours, or educational aspirations. While the impacts were generally small, 
the combination of impacts on attitudes and grades (especially core grade point 
average) suggests that service-learning is having a positive influence on school 
performance while students are in the program. 1 



1 For the purposes of this study, impacts are considered statistically significant if they are significant at 
the .05 level or higher. However, we will report and discuss impacts that are "marginally significant" (that is, 
significant at the .10 level) though they are not considered as reliable as the results with higher levels of 
significance. 
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• There were no statistically significant impacts for the participants as a whole on the 
measures of social or personal development, including communications skills, work 
orientation, or involvement in risk behaviors. However, there was a significant 
positive impact on arrests and teenage pregnancy for middle school participants, and a 
marginally significant impact on teenage pregnancy for the participants as a whole. 

The fact that there was an impact on two risk measures for middle school students 
suggests that involvement in well-designed service learning may play a role in 
reducing some risk behaviors among younger students. Similarly, the finding on 
teenage pregnancy, when coupled with results from other studies, suggests that while 
service alone may not dramatically reduce risk behaviors, service may contribute to 
the effectiveness of more comprehensive programs targeted to reducing risk behaviors 
among school-aged youth. 

• Analysis of impacts among different populations in the study also indicate that the 
impacts of service-learning were shared relatively equally by a wide range of youth 
(white and minority, male and female, educationally and economically disadvantaged, 
etc.). While some groups showed stronger impacts in one area or another (for 
example, minority students showed relatively strong impacts on grades), there were no 
consistent differences in impacts among the subgroups, and most of the positive post- 
program impacts were shared across the board. 



Participant Impacts at Follow-Up 

While there were a number of positive impacts at post-program, by the time of the follow-up 
study in Spring, 1997, most of the post-program impacts had disappeared. For the participant group 
as a whole, the only impacts evident at follow-up were marginally significant positive impacts on 
service leadership, school engagement, and science grades. The follow-up data also indicate a decline 
in English grades for participants, though the average English grades for participants remained higher 
that those of comparison group members at the time of follow-up. 

• In general, students from the high school programs showed a stronger pattern of 
impacts at follow-up than students from the middle schools. High school students 
showed positive, statistically significant impacts on service leadership and science 
grades, and marginally significant impacts on school engagement and volunteer hours. 
For the middle school students the only significant impact at follow-up was a 
marginally significant positive impact on arrests. 

• Follow-up impacts were also significantly stronger for participants who had continued 
their involvement in organized service activities during the follow-up year when 
compared to those for students who reported no organized service involvement in the 
follow-up period. Students who continued their involvement in organized service 
show positive impacts on measures of service leadership, service hours, and school 
engagement, as well as marginally significant impacts on involvement in service, 
college aspirations, and consumption of alcohol. For several of these measures, the 
gains for "repeaters" were significantly larger than those for students who did not 
continue their involvement in service during the follow-up year. 
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Overall, the follow-up data provide little evidence that one-time participation in even a well- 
designed service-learning program is likely to have substantial long-term participant impacts. The 
data does suggest that students who continue their involvement in service are significantly more likely 
to continue to experience the benefits of participation. 



Student Assessments of the Program Experience 

When asked directly about their program experience (through post-program surveys and 
interviews), program participants gave high marks to their service-learning experience: 

• More than 95 % of the program participants reported that they were satisfied with their 
community service experience and that the service they performed was helpful to the 
community and the individuals they served. 

• 87 % of the participants believed that they learned a skill that would be useful in the 
future, and 75% said that they learned more than in a typical class. 

• 75% reported developing a good personal relationship through service, generally with 
other students or a service beneficiary. 

• Over 90% felt that students should be encouraged (though not required) to participate 
in community service. 



Service in the Community 

The services provided by the Learn and Serve programs were highly rated by the community 
agencies, schools, hospitals, and other agencies where students provided assistance. 

• 99% of the agencies rated their overall experience with the local Learn and Serve 
program as "good" or "excellent." 

• 97% of the agencies indicated that they would pay at least minimum wage for the 
work being done, and 96% reported that they would use participants from the 
program again. 

• 90% of the agencies indicated that the Learn and Serve participants had helped the 
agency improve their services to clients and the community, and 68% said the use of 
the participants had increased the agency’s capacity to take on new projects; 

• 66% reported that the experience had increased the agency’s interest in using student 
volunteers; 
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• 56% said that participating in the program had produced new relationships with public 
schools, and 66% said that it had fostered a more positive attitude towards working 
with the public schools; and 

• 82% reported that the Learn and Serve program had helped to build a more positive 
attitude towards youth in the community. 



Integrating Service into Schools 

The service learning programs in the study were strongly supported by administrators and 
fellow teachers on average, and the large majority of programs appear likely to continue to operate 
after the end of their Learn and Serve grants. However, despite the general support for service- 
learning, few of the sites engaged in organized efforts to expand the use of service within the school 
or the district, and during the two years in which the sites were followed, there was no significant 
increase in the proportion of teachers using service-learning or measurable change in teaching 
methods or school climate. 



Return on Investment 

Based on an analysis of program costs and the value of the volunteer services provided by 
program participants, it is clear that the dollar benefits of well-designed service-learning programs 
substantially outweigh the costs. On average, participants in the programs in the study produced 
services valued at nearly four times the program cost during the 1995-96 program year. While the 
dollar value of participant gains in attitudes or academic performance cannot be calculated, they do 
represent an additional benefit. The net result is a substantial return on the public investment. 



Conclusions 

The data from the evaluation confirm that well-designed, school-based service-learning 
programs have a positive impact on young peoples’ civic and educational attitudes and school 
performance while also meeting important community needs. While the impacts on participants tend 
to fade in the year after initial participation, students who continue their involvement in organized 
service do continue to show positive effects over the longer-term. When compared to other studies, 
the results from this evaluation also lend support to the argument that "well-designed", "fully- 
implemented" service-learning programs are more likely to produce positive impacts on participating 
youth. As such, the findings highlight the importance of the Corporation and the states continuing 
their emphasis on improving the quality of local service-learning programs. The more that Learn and 
Serve programs begin to resemble the more intensive, fully-implemented service-learning efforts in 
this study, the more likely those programs will meet the goals of the national community service 
legislation. 
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Chapter One 

INTRODUCTION 



This report presents the final results from the national evaluation of the Learn and Serve 
America School and Community-Based Programs. The evaluation was based on a study of seventeen 
school-based service initiatives that were selected to represent mature, fully-implemented service- 
learning programs. The report includes five major areas of analysis: an analysis of the short-term 
("post-program") impact of the Learn and Serve programs on participants, based on surveys and 
school record information for approximately 1,000 middle and high school students from the 1995-96 
school year; an analysis of longer-term results based on a follow-up study conducted in 1997; an 
analysis of the service provided by program participants to their communities; an analysis of the 
integration of service in the participating schools; and an assessment of the program’s return on 
investment. 



The Learn and Serve Program 

In 1993, the National and Community Service Trust Act (P.L. 103-82) established the Learn and 
Serve America School and Community-Based Programs to support school and community-based 
efforts to involve school-aged youth in community service. The primary purpose of Learn and Serve 
is the involvement of school-aged youth in programs and classroom activities that link meaningful 
service in the community with a structured learning experience (i.e., service-learning). The goals of 
the program are to help young people develop as responsible citizens, improve their academic skills, 
and develop as individuals, while helping to meet "the unmet human, educational, environmental, and 
public safety needs of the United States." Learn and Serve is also designed to promote the integration 
of service-learning in schools and academic curriculum, and to promote the delivery of needed 
services in the community.' 

The Learn and Serve program is administered by the Corporation for National Service. The 
program is funded through grants to states and national organizations, and through them to individual 
school districts, schools, and community organizations. In 1994-95, the first year of the program, the 
Corporation awarded approximately $30 million in grants supporting over 2,000 local efforts 
involving over 750,000 school-aged youth. 



1 The Learn and Serve program is the successor to the Serve-America program, which was established under 
the original 1990 National and Community Service Act (P.L. 101-610). The 1993 legislation creating Learn and 
Serve modified and expanded Serve-America by creating separate funding streams for school and community-based 
programs, and by increasing the emphasis on service-learning. Funding for the program also increased from 
approximately $16 million annually for Serve-America to approximately $30 million for Learn and Serve. The 1993 
National and Community Service Trust Act also established the Learn and Serve Higher Education program, which 
supports the participation of postsecondary students in service, and the AmeriCorps program, a full-time national 
service corps, which provides stipends and educational benefits to individuals who provide up to two years of full- 
time volunteer service. 
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Overview of the Learn and Serve Evaluation 

In 1994, the Corporation for National Service selected Brandeis University’s Center for Human 
Resources and Abt Associates to conduct an evaluation of the Learn and Serve School and 
Community-Based Programs. The Learn and Serve evaluation was designed to address four 
fundamental questions: 

1 . What is the impact of program participation on program participants in terms of 
citizenship- related attitudes, involvement in community service, increased educational 
attainment, improved life skills, and reduced risk behaviors? 

2. What are the institutional impacts of Learn and Serve programs on participating schools 
and community agencies— in terms of the establishment of permanent service-learning 
opportunities, the incorporation of service-learning into mainstream learning, and the 
development of new partnerships between schools and the community? 

3. What impacts do Learn and Serve programs have on their communities— in terms of the 
specific accomplishments of service programs, the impact on service beneficiaries, and the 
impact on the broader community in terms of increased volunteerism or collaboration 
among schools and community agencies? 

4. What is the return (in dollar terms) on the Learn and Serve investment? 

To answer these questions, the evaluation examined middle and high school Learn and Serve 
programs in seventeen schools across the country using a variety of quantitative and qualitative 
methods. Major elements of the evaluation included: 

1. A participant impact study, based on pre- and post-program and follow-up surveys of 
program participants and comparison group members, as well as analysis of school record 
data and on-site student interviews; 

2. An institutional impact study that combined on-site interviews with program coordinators, 
school administrators, faculty, students, and host agency representatives with school-wide 
surveys of teachers in the participating schools to examine broader use and attitudes 
towards service-learning; 

3. A community impact analysis based on a series of telephone surveys of host agency 
(service site) administrators in all seventeen sites, collecting information on program 
accomplishments, service quality, and the estimated dollar value of the services provided; 

4. Analysis of return on investment based on the data collected through the host agency 
surveys (estimated value of service) and program financial data. 

The focus for the evaluation was the 1995-96 school year, with student and teacher follow-up 
surveys taking place in Spring 1997. 
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The Evaluation Study Sites 

The basis for this evaluation was the experience of seventeen middle and high school Learn and 
Serve programs in nine states around the country. 2 The programs are described in Exhibit 1.1 at the 
end of this chapter. The programs were selected through a purposive sampling process aimed at 
identifying well-established, fully-implemented service-learning programs, defined as those programs 
that had been in existence for more than one year (to eliminate obvious start-up problems) and that 
reported higher than average hours of service and regular use of written and oral reflection. The goal 
in this process was to select well-designed programs that represented a more intensive, higher quality 
service-learning experience than average. All the sites in the evaluation were school-based initiatives 
and linked to a formal course curriculum, either as part of a core subject (for example an English or 
social studies class) or an elective course. The evaluation sites were selected from a pool of 
approximately 210 middle and high school service-learning programs that had been randomly selected 
and contacted as part of the site selection process. 3 

The decision to focus the evaluation on more intensive, fully-implemented, school-based service- 
learning programs has important implications for understanding the evaluation findings. The Learn 
and Serve evaluation was designed to build on the recently completed Serve-America evaluation 
(which was based on a representative sample of programs) by focusing on a subset of programs that 
met basic quality criteria for effective service-learning, rather than on a representative cross-section of 
all Learn and Serve programs nationally. Based on the information gathered for the site selection 
process, these programs represented approximately 15% of the Learn and Serve programs operating 
in 1994-95. As such, this evaluation was not designed to address the average impact of aU Learn 
and Serve programs, but rather to identify the impacts that can be reasonably expected from 
mature, fully-implemented, school-based service-learning efforts. In that regard, the programs in the 
evaluation should be considered as representing the upper tier of Learn and Serve programs, and the 
evaluation should be seen as indicative of the potential impacts for service-learning as programs 
mature and implementation improves throughout the system. 4 



2 The nine states are California, Florida, New Mexico, New York, North Carolina, Ohio, Pennsylvania, Texas, 
and Wisconsin. 

3 The site selection process is described in detail in Appendix A. Briefly, the evaluation randomly selected 10 
states from a weighted pool of Learn and Serve grant recipients and then contacted a sample of subgrantees in each 
state to confirm or collect information on the local Learn and Serve programs. In all, information was collected on 
210 local programs. The information on those programs, in turn, was then used to identify the pool of programs that 
met the selection criteria. The evaluation team contacted those programs and solicited their participation in the 
evaluation. Originally, 18 programs agreed to participate in the evaluation. One program was later dropped because 
of difficulties in collecting participant information. Elementary schools were not included in the evaluation, primarily 
because of the differences and difficulties in assessing impacts for that age group. 

* There were a number of reasons for focusing the evaluation on this subset of programs. At the time of the 
evaluation design, Abt Associates and Brandeis University were completing an evaluation of the Serve-America 
program, Learn and Serve’s predecessor, based on a representative sample of sixteen program sites. Given the 
similarities between Serve-America and Learn and Serve programs, it was felt that an evaluation based on a similar 
methodology would provide little additional information to aid the Corporation’s policy-making and program 
oversight. At the same time, there were indications in the Serve-America study that program quality and intensity 
were factors that affected program impact (See Alan Melchior and Larry Orr, Final Report: National Evaluation of 
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Though representing a relatively select group of programs in terms of intensity and 
implementation, the seventeen programs included in the study did vary substantially in their 
organization and structure, reflecting much of the broader diversity among Learn and Serve programs 
around the country: 

• Ten of the programs were high school programs and seven serve middle school students. 

• Ten were integrated into academic classes; seven were structured as stand-alone, elective 
service-learning courses. Nine of the programs were part of a school-wide service or 
service-learning strategy. 

• Four of the Learn and Serve programs were integrated into special programs for at-risk 
youth. Three of the programs (including two of those targeted to at-risk youth) took place 
within alternative school settings. 

• Eight of the programs took place in urban settings, five were primarily suburban, and four 
took place in rural areas. 

• Overall, the Learn and Serve programs in the study sites ranged in size from 21 students 
to over 400 participants in the school-wide efforts. Direct service hours ranged from an 
average of 3 hours per participant to 196, with an average of 77 hours across the sites. 

The students in these programs also represented a diverse group of young people. It is 
important to recognize that while the programs were selected to represent a particular level of 
implementation, they were not selected based on their participant characteristics. As a result, they 
included a diverse group of young people in terms of age, ethnicity, socioeconomic background, and 
prior experience with service-learning (see Exhibit 1 .2): 

• 29% of the participants were in middle school (grades 6-8) and 71 % in high school (grades 
9-12). The largest single group was high school seniors (35%). 

• 60% were female, 40% male. 

• 58% of the students were White, 17% African-American, 19% Hispanic; 6% indicated that 
they were Asian, Native American, or multicultural; 95% came from English-speaking 
homes. 



Serve-America , Cambridge, MA: Abt Associates, December, 1995). The decision was made, as a result, to focus the 
Learn and Serve evaluation on programs that were well-established and represented fully-implemented service-learning 
models. 

The decision to focus on programs that were linked to a formal course curriculum was based on a similar set of 
considerations. Approximately 28% of the local programs contacted in the course of the site selection process 
characterized themselves as "service only" programs— generally after-school community service clubs. Given the 
emphasis on service-learning rather than service per se in the 1993 legislation, it was decided to focus the evaluation 
on those school-based programs that had a service-learning focus. 

Finally, the decision to limit the evaluation to school-based efforts (and exclude community-based programs 
from the study) reflected the common trade-off between the scope and reliability of the study and available resources. 
Given the resources for a limited number of sites, it was decided to focus the study on school-based programs (which 
represent the major focus of the legislation and federal funding) rather than split the sample among school and 
community-based initiatives. 
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• 38% were economically disadvantaged and 30% were identified as educationally 
disadvantaged. 5 

• 29% had been involved in some form of self-reported delinquent behavior (been in a fight, 
used a weapon, hurt someone badly) during the past 6 months at baseline. 

• 45% had been involved in a service-learning class in a prior year. 6 



Organization of the Report 

The remainder of the report presents the results of the evaluation. Chapter Two provides an 
overview of the programs and the service experience in the evaluation sites. Chapter Three then 
presents data on the post-program participant impacts, based on the analysis of survey and school 
record data. Chapter Four presents a similar analysis of participant impacts at follow-up. Chapter 
Five then presents the students’ perspective on their service experience, using a mix of interview and 
survey data. Chapter 6 discusses the services that program participants provided to their communities 
and provides an assessment of the service quality and impact based on surveys of staff at the local 
service sites. Chapter 7 examines the institutional impacts of Learn and Serve on participating 
schools with a particular focus on the institutionalization and integration of service-learning. Finally, 
Chapter 8 presents the findings on the dollar return on investment for the Learn and Serve programs 
in the study. 

In a separate volume, appendices provide additional details related to the major findings in the 
report. Appendix A provides information on the site selection process. Appendix B reviews the 
methodology used in the participant impact analysis and includes information on the characteristics of 
the participants and comparison group members at baseline. Appendix C provides information on the 
measures used in the participant analysis. Appendix D provides statistical tables detailing the results 
of the overall post-program participant impact analysis, and Appendix E provides tables with results 
from the subgroup analysis of the post-program data. Appendix F provides tables with the follow-up 
participant impact results, and Appendix G provides data on the subgroup differences at follow-up. 
Appendix H presents the data on program "repeaters" and "non-repeaters" at follow-up — that is, 
students who were and were not involved in service during the follow-up year. Finally, Appendix I 
includes copies of the survey instruments used in the study. 



5 Data on economic and educational status were reported by the schools as part of the school record data 
collection. Economically disadvantaged was defined as students who were eligible for free and reduced cost lunch, 
JTPA, Food Stamps, AFDC, or other income tested programs. Educationally disadvantaged students were those who 
were reported as eligible for Chapter 1 or Special Education or who were more than two years below grade level in 
reading or had been retained in grade at least once. 

6 Figures are based on the 608 program participants in the analysis sample for the study. Information on 
participant characteristics are drawn from the baseline survey data and school record information. 
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Exhibit 1.2 

Participant Characteristics at Evaluation Sites 



Participant Characteristics |i| 


I All Participants 


lf| High School 


Middle School ;f 


Gender 








Male 


40% 


38% 


45% 


Female 


60% 


62% 


55% 


Race/Ethnicity 








White 


58% 


59% 


53% 


African-American 


17% 


8% 


39% 


Hispanic 


19% 


25% 


3% 


Asian 


2% 


3% 


0% 


Native American 


1% 


1% 


1% 


Multicultural 


3% 


3% 


3% 


Other 


0% 


0% 


1% 


English spoken at home 


95% 


95% 


96% 


Economically disadvantaged 


38% 


29% 


59% 


Educationally disadvantaged 


30% 


24% 


45% 


Involved in delinquent behavior in 








past 6 months 


29% 


25% 


38% 


Participated in a service-learning 








class in a prior year 


45% 


44% 


48% 


Number 


608 (100%) 


435 (72%) 


173 (28%) 



Source : Baseline survey of participants in the 17 evaluation sites (435 high school students, 173 middle school 
students). 
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THE PROGRAM EXPERIENCE 
IN THE EVALUATION SITES 



At the core of the Learn and Serve program is the idea of service-learning. As defined in the 
legislation, service-learning combines meaningful service in the community with a formal educational 
curriculum and structured time for participants to reflect on their service experience. Service-learning 
stands in contrast to traditional voluntarism or community service, which generally does not include 
reflection or links to any organized curriculum. 1 

As noted in Chapter One, the Learn and Serve evaluation is focused on sites that meet the basic 
set of criteria for high quality, fully-implemented service-learning. All of the sites involve students in 
higher than average service hours and all conduct regular reflection and writing. The programs are 
all school-based and linked to an academic curriculum. While the programs vary in structure and 
format, all offer a relatively intensive, hands-on involvement in service and an opportunity to 
"process" the service experience through formal and informal group discussions, journal writing, 
research papers, and group presentations. 

Exhibits 2.1 through 2.3 provide an overview of the characteristics of the service experience in 
the evaluation sites and help to give some shape to the service-learning experience of participants in 
the study. Exhibits 2.1 and 2.2 are based on information from the post-program participant surveys 
and service rosters; Exhibit 2.3 is based on information compiled during site visits. 

As Exhibit 2.1 shows, on average, students in the evaluation sites were involved in substantial 
hours of direct service. While the hours for individual programs varied widely, from an average of 3 
hours of direct service per student in one program to nearly 200 hours in another, the average for the 
evaluation sites as a whole was over 70 hours, more than two times the median among the national 
sample used to select the evaluation sites. 2 For most students, service was in educational or human 



1 The National and Community Service Trust Act defines service-learning as a method: "(A) Under which 
students or participants learn and develop through active participation in thoughtfully organized service that— (i) is 
conducted in and meets the needs of a community; (ii) is coordinated with an elementary school, secondary schools, 
institutions of higher education, or community service programs, and with the community; and (iii) helps foster civic 
responsibility; and (B) that (i) is integrated into and enhances the academic curriculum of the students, or the 
educational components of the community service program in which the participants are enrolled; and (ii) includes 
structured time for the students and participants to reflect on the service experience." (U.S. Code Title 42, Section 
12511) 

2 The average hours per student are for the program period, generally a school year (two of the seventeen 
programs were one semester only). It is worth noting that several of the programs ultimately reported fewer service 
hours than were indicated in the program descriptions used for site selection. In some cases the differences represent 
changes made in the program during the course of the 1995-96 school year. In most cases, however, the difference 
reflects the difficulties of defining and measuring service hours in an integrated program (for example, preparation 
time for a public event can count as classroom or service time). The service hours reported here are those hours in 
which students were actually performing/delivering services. 
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Exhibit 2.1 

Characteristics of Service-Learning Experience 
in Evaluation Study Sites 



Characteristic . 


Overall 


High 

School 


Middle 

School 


Average Hours of Direct Service Per Participant 


77 hrs. 


86 hrs. 


53 hrs. 


Types of Service” 








Education 


65.5% 


67.8% 


56. 1 % 


Human Need 


59.2% 


63.9% 1 


46.8% 


Environment 


38.2% 


34.3% 


48.0% 


Public Safety 


23.0% 


22. 1 % 


25.4% 


Service activities were mostly: b 








Directly helping other people 


43.4% 


49.9% 


27.2% 


Indirectly helping other people 


21.5% 


15.6% 


36.4% 


Both types of activities in equal amounts 


32. 1 % 


30.8% 


35.3% 


Work was conducted: 








As part of a team 


28.2% 


24.0% 


38.4% 


Individual assignment 


11.8% 


12.6% 


9.9% 


Combination of individual and team assignments 


60.0% 


63.4% 


51.7% 


Kept a journal or diary as part of community service project 


44.4% 


50.8% 


28.1% 


Community service project included time in class to talk about 


75.9% 


77.7% 


71.3% 


service experience 









Sample Sizes: Overall analysis sample: 608; high school: 435; middle school: 173. Sample sizes for individual 
items vary slightly due to item nonresponse. 

* Percentages do not sum to 100 because participants engaged in multiple service activities. 
b Direct service activities were those in which students had direct, face-to-face contact with service recipients. 

Indirect activities included activities such as fundraising, food and clothing drives, recycling, or park improvement. 

services-related projects— for example, tutoring or working as a teacher’s aide, working in a nursing 
home or homeless shelter. 

The service also involved at least some hands-on, face-to-face experience with recipients. The 
large majority of students (75.5%) had at least some direct contact with service recipients, meeting 
students or senior citizens face-to-face. Fewer than 25% of the participants were in programs where 
service projects were solely indirect in nature (such as food drives or park clean-up). Most students 
(60.0%) also experienced a combination of individual and group service assignments— each of which 
has its strengths (for example, group projects can teach teamwork, individual assignments can help 
reinforce personal responsibility). Less than one third of the students did service only in a group, and 
less than 12% had only individual placements. 
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Service also involved reflection. Seventy-six percent of the participants reported that their 
classes included time set aside to discuss their service experiences, and 44% reported keeping a 
journal. Many of the programs also used other forms of written reflection (essays, research papers, 
presentations) not captured by the survey questions. 



Exhibit 2.2 

Characteristics of Service Experience 

(Percent responding "very often" or "fairly often”) 



Characteristics 


Overall 

(Percent) 


High School 
(Percent) • 


Middle School 
(Percent) 


Had real responsibilities 


79.4 


81.1 


75.1 


Felt I made a contribution 


78.2 


81.1 


71.0 


Did things myself instead of observing 


66.4 


72.3 


51.8 


Had a variety of tasks to do at site 


69.9 


70.7 


68.0 


Had freedom to develop and use my own ideas 


65.1 


68.3 


57.4 


Had freedom to explore my own interests 


61.7 


63.0 


58.6 


Adults at site took a personal interest in me 


59.0 


62.4 


50.3 


Had challenging tasks 


59.0 


58.7 


59.8 


Made important decisions 


59.1 


56.9 


64.5 


Discussed my experiences with teachers 


52.6 


55.1 


46.2 


Needed more help from my supervisor 


11.4 


9.8 


15.4 


Adults criticized me or my work 


7.5 


5.8 


11.9 


Discussed my experiences with family and 
friends 


60.6 


63.9 


52.7 



Sample Sizes: Overall: 608; High School: 435; middle school: 173. Sample sizes for individual items vary slightly 
due to item nonresponse. 

Exhibit 2.2 presents additional background on the students’ service experience, based on a 
second set of questions in the post-program survey, and confirms a relatively high quality experience 
for the majority of program participants. Over 70% of the participants felt they had real 
responsibilities, did things themselves, had a variety of tasks, and made a contribution "very often" or 
"fairly often." The majority of participants also felt they made important decisions, discussed their 
experiences with teachers, family, and friends, had the freedom to develop their own ideas, and 
gained the personal interest of the adults with whom they worked. It is important to note that middle 
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and high school participants report similar experiences. This contrasts sharply with the findings in 
the Serve-America evaluation, in which the middle schools clearly presented a much less intensive, 
hands-on program experience than either the high school programs in that evaluation or the middle 
and high school programs in this study. 3 

Finally Exhibit 2.3 suggests some of the ways in which programs organized their service- 
learning and integrated service into academic and elective courses. As is clear from the snapshot 
descriptions, programs organized their activities in many different ways, but each built in substantial 
time for reflection and opportunities to make connections to the curriculum. 

Taken together, the data on the experience of program participants in the evaluation sites helps 
to set the context for understanding the participant impacts. Though the evaluation sites did vary 
among themselves in terms of the intensity and structure of the service-learning experience, as a 
group they represent a relatively strong effort to implement the ideal of service-learning. 



29 

3 See Melchior and Orr, Final Report: National Evaluation of Serve-America , Chapter 4. 
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Exhibit 2.3 

Overview of Selected Evaluation Sites 

The SITES program at North Olmsted High School (Ohio) combines English, social studies, and 
service-learning into a single half-day block of classes. Two days a week students are in class for the 
full 3-period block. Three days a week students are in class for one period and work at their service 
sites for the other two periods, averaging 4-5 hours of service per week. In the social studies classes, 
the class links academics and service through a series of research papers and presentations on topics 
related to students’ service sites (for example, students working in hospitals or nursing homes 
researched the national health care debate). The English class is linked to service through its literature 
readings (for example, The Jungle and Walden II), which focus on the relationship of "man and 
society." 

In the Social Issues program at Taos High School (New Mexico), students identify, study, and 
address local issues through small group projects which range from training as drug education 
counselors for the elementary school to sponsorship of a student/police basketball game as part of an 
anti-violence campaign. Students write about and discuss their projects and their role in the 
community through weekly reflection exercises designed by the course instructors. In 1995-96, the 
teachers added an "introductory service- learning project," in which students performed service for 
family members and friends and wrote about the experience, as a way of introducing the service 
concept to students with no prior experience as volunteers. 

East Scranton Intermediate School (Pennsylvania) has a school-wide service philosophy and 
developed an interdisciplinary 8th grade class focused on service-learning. Their major service activity 
is focused on a local hospital, where students work in a variety of departments. Students also work 
together on a variety of small group projects tied to academic subjects. Students participate in service 
three out of every six afternoons, totalling more than 200 hours over the course of the school year. 

At Wakulla Middle School (Florida), service-learning is integrated into the alternative education 
program for at-risk students. Every other week students in the at-risk program, along with high 
achieving students, work for half a day with staff from the Park and Recreation Department to 
revitalize a neighborhood park. Students work in small groups on tasks that reinforce social and 
behavioral skills (for example, working in groups), as well as some academic skills (measuring, reading 
instructions, etc.). Each service session is followed by an organized group discussion. Teachers in the 
alternative education program then use the park experience over the year in illustrating lessons in the 
classroom. 

At Scotia High School (New York), the GIVE program is an elective service-learning course that 
operates on a quarterly basis. Students can enroll for one or more academic quarters — up to a 
maximum of sixteen. Each student attends the GIVE class once each week (the class itself is offered 
three times a week — before, after, and during school — so that every student can attend), and provides 
two or more hours of direct service every week. Service activities range from one-to-one support for 
elderly residents (students make daily calls to check on their elderly partner), to volunteer work at a 
range of local human service agencies. Reflection takes place through the weekly class discussions and 
a weekly "reflection document" (similar to a journal) which is reviewed by the teacher and which 
students share to spark discussions in class. 
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PARTICIPANT IMPACTS 



The primary goal of the Learn and Serve program is to help young people develop as 
responsible citizens, improve their academic skills, and develop as individuals through involvement in 
meaningful service linked to structured learning activities. Because of this, three basic questions 
guide the participant impact evaluation: 

1. What was the impact of service-learning on participants’ civic development? Did service- 
learning help to build students’ understanding of their communities, their sense of social 
responsibility, and their commitment to community involvement? 

2. What was the impact on educational development and academic performance? Did 
service-learning increase students’ engagement in school, school attendance, and/or 
academic performance? 

3. What was the impact of service-learning on students’ personal and social development? 
Did service-learning help strengthen students’ life skills (such as communications skills, 
work orientation, and career awareness), and did it lead to a reduction in involvement in 
risk behaviors? 

To address these questions, the evaluation examined participant impacts at two points in time. 
First, the evaluation examined participant impacts at the end of the 1995-96 program year to identify 
short-term, "post-program" impacts — those impacts that were evident immediately following program 
participation. The evaluation then conducted a follow-up study in the spring of 1997 to examine the 
longer-term impacts of program participation. In both cases, the assessment of participant impacts 
was based on a combination of participant surveys (at baseline, post-program, and at follow-up) and 
data drawn from school records. Finally, the evaluation team also collected information on 
participants’ responses to their programs and their service experiences through the post-program 
surveys and through interviews conducted with the students at the end of the 1995-96 program year. 

The purpose of this chapter is to report on the post-program impacts on program participants. 
(Findings on the impacts at follow-up are reported in Chapter 4, and participant assessments are 
discussed in Chapter 5,) Two major findings are reported here: 

1 . Based on the data from the 1995-96 school year, the Learn and Serve programs in this 
study had a positive, statistically significant post-program impact on measures of civic 
attitudes and behavior and on several measures of educational attitudes and school 
performance. There were no consistent impacts on measures of personal and social 
development— though there are scattered and marginal impacts that do suggest that service- 
learning can be beneficial in this area as well. 1 



1 For the purposes of this study, impacts are considered statistically significant if they are significant at the .05 
level or higher. However, we do report and discuss impacts that are "marginally significant" (that is, significant at 
the .10 level) though they are not considered as reliable as the results with higher levels of significance. 
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2. Post-program impacts were shared relatively equally by a wide range of participating 
youth (white and minority youth, males and females, educationally and economically 
disadvantaged youth, etc.) While some groups showed stronger impacts in one area or 
another (for example, minority students showed relatively strong impacts on grades), there 
were no consistent differences in impacts among the subgroups, and most of the positive 
post-program impacts were shared across the board. 

This chapter reviews the short-term participant impacts in the seventeen evaluation sites. It 
begins by providing a brief overview of the data sources and study’s approach to measuring post- 
program impacts. It then reviews the post-program impact findings for the participant group as a 
whole and for the two primary subgroups: middle school and high school participants. Finally, the 
chapter examines the differences in impacts among other major subpopulations in the study. 2 



Measuring Post-Program Participant Impacts 

To measure the short-term, "post-program" impact of the Learn and Serve programs in the 
study on program participants, the evaluation administered pre- and post-program surveys to a sample 
of participants and a comparison group in each of the evaluation sites and analyzed school record 
information for both groups. In the smaller sites, all of the program participants were included in the 
evaluation; in the larger, school-wide sites, one or more classrooms were selected for survey 
purposes. 3 Comparison group members were generally students in similar types of classes in the 
same school (e.g. core academic or elective), matched as closely as possible with participants in terms 
of demographic characteristics (age, gender, race/ethnicity, etc.) and academic status. In several 
instances where program activities were school-wide, the comparison groups were drawn from similar 
schools in the same or neighboring communities. Altogether, the analysis sample for the post- 
program impacts includes 1 ,052 students for whom both baseline and post-program survey and school 
record data are available. Of these, 608 were program participants and 444 were comparison group 
members; 733 of the students were high school-aged and 319 were middle school students. Exhibit 
3.1 provides an overview of the analysis sample. The analysis sample and impact estimation methods 
are also discussed in more detail in Appendix B. 

The surveys and school records used in the study incorporated over 20 different outcome 
measures, including measures of civic and social attitudes, involvement in volunteer activity, 
educational attitudes and performance, and measures of involvement in risk behaviors. The measures 
reported in the study are listed in Exhibit 3.2 and described in Appendix C. 



2 This chapter includes a substantial revision of the analysis of post-program participant impacts first reported 
in the evaluation’s interim report (Melchior et al. National Evaluation of Learn and Serve America, School and 
Community-Based Programs, Interim Report, April, 1997), with some changes in that report’s conclusions based 
on use of a different set of statistical techniques. The findings in this report should be considered as 
superseding those of the interim report. 

3 Sample classes were selected in three of the sites: Sierra Ridge Middle School, Futures Academy, and 
Caprock High School. In all three sites an effort was made to identify classes that met the overall selection criteria 
for the study (e.g. in existence for more than one year, higher than average hours, regular reflection and writing). At 
Caprock, the study focused on 9th grade classes to minimize the effects of prior participation in the program. 
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Exhibit 3.1 

Overview of Post-Program Impact Analysis Sample 





Participant Group 


: : Comparison Group 


Total 


Middle School 


173 


146 


319 


High School 


435 


298 


733 


Total 


608 


444 


1,052 



Source-. Participants and comparison group members with matched pre- and post-program surveys and school record 
information. 



In general, program impacts were estimated by comparing the average outcomes for program 
participants with those of comparison group members after making adjustments through a regression 
formula for differences in both baseline scores and the baseline characteristics of the two groups. By 
taking into account the initial differences between participants and comparison group members, the 
regression analysis allows us to isolate the estimated impacts of service-learning from those that might 
be caused by other differences between the two groups. 

In the course of the evaluation, two different statistical techniques were used to estimate the 
program impacts. The initial analysis, reported in the evaluation’s interim report, used an analysis of 
covariance approach which adjusted for differences in baseline scores by including those scores as a 
separate variable in the regression formula. In this approach, baseline scores are treated like other 
covariates in the regression formula (for example, age, gender, etc.). The "outcome" that is 
estimated is the post-program score as adjusted for differences in both baseline scores and participant 
characteristics. This was the method used in the earlier evaluation of Serve- America and has been 
commonly used in other evaluations of service-learning programs. 4 In this report, we refer to this 
approach as the analysis of covariance or ANCOVA model. 

In this report, we also use a second method that adjusts for differences at baseline by calculating 
the difference between post-program and baseline scores (that is, post-program score minus baseline 
score). In this case, the outcome that is estimated is this difference in pre- and post-program scores, 
and the analysis examines the difference in pre/post changes for participants and comparison group 
members. We refer to this approach as the difference-in-difference model (or DD model) in this 
report. 5 



4 See, for example, Fred Newman and Robert Rutter, "The Effects of High School Community Service 
Programs on Students’ Social Development," University of Wisconsin, Center for Educational Research, 1983. 

5 In point of fact, both approaches make use of analysis of covariance as a statistical technique. The terms 
Analysis of Covariance and Difference-in-Difference are simply used to distinguish the two approaches for the 
purposes of the study. 
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Exhibit 3.2 

Outcome Measures Used in the Evaluation 



Civic/Social Attitudes 


• 


Personal and Social Responsibility (including Welfare and Community Involvement 
Subscales) 


• 


Acceptance of Cultural Diversity 


• 


Service Leadership 


• 


Civic Attitudes-Combined Scale (Combined scores from Personal and Social Responsibility, 
Cultural Diversity, and Service Leadership) 


Volunteer 


Behavior 


• 


Involvement in any Volunteer Activity in Past 6 Months 


• 


Estimated Hours of Volunteer Service in Past 6 Months 


Educational Impacts 


• 


Educational Competence 


• 


School Engagement 


• 


Course Grades (English, Social Studies, Math, Science) 


• 


Overall Grade Point Average 


• 


Core Grade Point Average 


• 


Failed 1 or More Courses 


• 


Days Absent 


• 


Days Suspended 


• 


Educational Aspirations (Graduate 4 Year College) 


■ • 


Homework Hours (3 or More Hours per Week) 


Social Development 


• 


Communications Skills 


• 


Work Orientation 


Involvement in Risk Behaviors 


• 


Consumed any Alcohol in Past 30 Days 


• 


Used Illegal Drugs in Past 30 Days 


• 


Arrested in Past 6 Months 


• 


Ever Pregnant or Made Someone Pregnant 


• 


Fought, Hurt Someone, or Used Weapon in Last 6 Months 



Among statisticians today there is some debate as to the relative strengths and weaknesses of the 
two approaches and when it is most appropriate to use each. In general, it is now believed that the 
ANCOVA model, by adjusting for differences in baseline scores through the regression calculation, 
tends to underadjust for differences at baseline. As a result, when program participants score more 
highly at baseline (which is the case for most measures in this study), the ANCOVA model is likely 
to overstate program impacts. The difference-in-difference model, which adjusts for differences at 
baseline by simply subtracting baseline from post-program scores is now thought to provide a more 
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unbiased estimate. 6 For this report, because of the previous work using the ANCOVA model, we 
are providing results from both approaches. However, generally the evaluation will rely on the 
results of the difference-in-difference model as the more "conservative" estimate of program impacts. 
Appendix B provides a detailed technical discussion of the two approaches and the differences in the 
nature of the estimates they produce. 7 



Post-Program Impacts on Participants 

Exhibit 3.3 presents an overview of the post-program participant impact findings for the 
participant group as a whole, and separately for participants in high school and middle school 
programs. Pluses and minuses in the exhibit indicate a positive or negative impact. There were no 
statistically significant negative impacts found among the post-program impacts in the study. 8 



Impacts on Civic/Social Attitudes 

Based on the results of the pre- and post-program surveys, the Learn and Serve programs in the 
study had a small but statistically significant positive impact on the civic attitudes of participants. As 
the first section of Exhibit 3.3 shows, Learn and Serve participants showed positive, statistically 
significant post-program impacts on three of four measures of civic development using the difference- 
in-difference approach: acceptance of cultural diversity, service leadership, and the combined measure 
of civic attitudes. Only the social and personal responsibility scale failed to show a significant 
impact. When the analysis of covariance approach is used, the impacts are even broader, with all the 
measures of civic attitudes showing strongly significant positive impacts. 

When high school and middle school students are looked at separately, both high school and 
middle school students show a broad array of impacts under the analysis of covariance model, with 
positive and statistically significant impacts across most of the measures of civic attitudes. With the 
difference-in-difference approach, the high school impacts are still relatively strong, with significant 
impacts on service leadership and the combined civic attitudes scale and a marginal impact on 
attitudes towards diversity. Middle school students, in contrast, show some gains in the measures of 
civic attitudes under the difference-in-difference model, but none are statistically significant. 



6 It is worth noting that where participant scores are lower at baseline, the ANCOVA model tends to bias 
impacts downward — to underestimate impacts. In both cases, this is because the regression formula only 
partially adjusts (i.e. underadjusts) for the differences at baseline. 

7 We are greatly indebted to Dr. Christopher Winship for his help in working through the differences between 
the two models. Two publications that provide excellent discussions of these issues are: Charles Judd and 
David Kenny, Estimating the Effects of Social Interventions (Cambridge: Cambridge University Press, 1981), 
especially Chapter 6; and Paul Allison, "Change Scores as Dependent Variables in Regression Analysis," 
Sociological Methodology , v. 20 (1990), 93-114. 

8 The results of the impact analyses are displayed in detail in Appendix D. 
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Exhibit 3.3 

Summary of Post-Program Participant Impacts 





ifft AH’: Participants 


SHigh School 


Middle School 








^^Participants ; 


Participants 






Difference 




Difference 




Difference 


. .. • . 


Analysis of 


-in-; : 


• Analysis of 


-in- ; 


Analysis 'of - - 




Characteristic > 


l|: Covariance 


Difference 


Covariance 


Difference : 


x Covariance 


Difference 


Civic/Social Attitudes 


Personal and Social 
Responsibility 














Social Welfare Subscale 


+ + + 




+ + 




+ + 




Community Involvement Scale 
Total Personal and Social 


+ + + 




+ + + 




+ + + 




Responsibility Scale 


+ + + 




+ + + 




+ + + 




Acceptance of Cultural Diversity 


+ + + 


+ + 


+ + + . 


+ 


+ 




Service Leadership 


+ + + 


+ + + 


+ + + 


+ + + 


+ + + 




Civic Attitudes-Combined Scale 


+ + + 


+ + + 


+ + + 


+ + + 


+ + + 




Volunteer Behavior 


Volunteered for a Community 
Organization or Got Involved in 
Other Community Service in 
Last 6 Months 


+ + + 


+ + + 


+ + + 


+ + + 


+ + + 


+ + + 


Average Hours Doing Volunteer 
Work or Community Service in 
Last 6 Months 


+ + + 


+ + + 


+ + + 


+ + + 


+ 




Educational Impacts 


Educational Attitudes 














School Engagement 


+ + + 


+ + + 


+ + + 


+ + + 


+ + 




Educational Competence 


+ 




+ 








Course Grades 














English 

Math 


+ + + 


+ + + 


+ + 


+ + 


+ + + 




Social Studies 


+ + + 








+ + + 


+ + 


Science 


+ + 


+ 




+ 


+ + 





a +/- indicates positive or negative impact. + is statistically significant at the 0.10 level; + + at the .05 level; 
+ + + at the .01 level (two-tailed test). 
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Exhibit 3.3 

Summary of Post-Program Participant Impacts (Continued) 



Characteristic 


All Participants 


High School 
Participants 


Middle School 
Participants 


Analysis of 
■Covariance 


Difference 

-in- 

Difference 


Analysis of 
Covariance 


Difference 

Difference 


Analysis of 
Covariance 


Difference 

-in- 

Difference 


Educational Impacts (continued) 


Overall/School GPA 


+ 












Core GPA 


+ + + 


-t 






+ + + 




Fail 1 or More Courses 


+ 




+ 


+ + 






Days Absent 














Suspended During Previous 
Semester 














Want to Graduate 4-Year College 
or Beyond 


+ + 




+ 








Homework Hours: 3 Hours or 
More Per Week 














Social Development/Involvement in Risk Behavior 


Psychosocial Maturity 
Communications Skills Subscale 
Work Orientation Subscale 






+ + 








Consumed Any Alcohol in Last 
30 Days 














Used Illegal Drugs in Last 30 
Days 














Arrested in the Last 6 Months 












+ + 


Ever Pregnant or Made Someone 
Pregnant 


+ 


+ 








+ 


Fought, Hurt Someone or Used 
Weapon in the Last 6 Months 






+ 









a +/- indicates positive or negative impact. + is statistically significant at the 0.10 level; + + at the .05 level; 

+ + + at the .01 level (two-tailed test). 

b Core GPA is calculated as the average of English, Math, Social Studies, and Science Grades. 
c Three (3) programs were excluded from the analysis of days absent due to incomplete reporting. 
d Seven (7) programs were excluded from the analysis of suspensions due to incomplete reporting. 

Source : Impacts on "All Participants" is based on analysis of baseline and post-program surveys of 608 program 
participants and 444 comparison group members (N = 1052). High school impact analysis is based on 435 high school 
participants and 298 comparison group members (N=733). Middle school analysis is based on 173 participants and 
146 comparison group members (N =319) 

I’ BEST COPY AVAILABLE 
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While in most instances this report will rely on the difference-in-difference approach for its 
interpretation, it is worth looking at both sets of results for the measures of civic and social attitudes. 
In those cases where the outcome being measured is likely to have influenced selection into the 
program (for example, where students with initially strong civic attitudes were likely to select into 
service-learning classes), the ANCOVA model is likely to overestimate impacts, but the difference-in- 
difference model is also likely to underestimate impacts. In essence, for civic attitudes, we need to 
consider the two models as providing high and low end estimates of the program impact, with the 
"true" impact somewhere in between. 9 

In practical terms, the fact that both the "conservative" difference-in-difference model and the 
more "generous" ANCOVA model show significant impacts on a number of the measures of civic 
attitudes gives substantial weight to the conclusion that the Learn and Serve programs are having an 
impact on participant attitudes. At the very least we know that, for the participant group as a whole, 
the programs are making a difference on attitudes concerning cultural diversity and service leadership, 
as well as on a combined measure of civic attitudes; it is possible, but somewhat less certain, that the 
service-learning programs are also affecting attitudes towards personal and social responsibility. 

While it is clear that Learn and Serve programs are having a positive impact on civic attitudes, 
it is also important to recognize that, whatever the method used in the analysis, these impacts are 
generally small: the largest, service leadership, shows only a 5% difference between participants and 
comparison group scores in the difference-in-differences approach; the difference is still only 8.4% 
under the analysis of covariance. In part, the relatively small size of the impacts reflects that fact that 
most young people begin with a fairly well-developed sense of civic responsibility (both participants 
and comparison group members scored highly at baseline on these measures). In that regard, service- 
learning programs might best be understood as strengthening or reinforcing students’ generally 
positive civic attitudes rather than building a positive set of attitudes from scratch. 

Finally, it is worth noting that, for high school participants and the participant group as a 
whole, the largest impact on civic attitudes was on the measure of service leadership, which was the 



9 The differences between the two models reflect different statistical assumptions about measurement error and 
the likelihood of regression toward the mean in the scores of the participant and comparison group. In essence, 
the ANCOVA model assumes that, in this instance, students may have felt unusually "civic minded" at the point 
in which they selected into a service-learning program, and that on average, their sense of civic mindedness was 
usually somewhat less. In that instance, the ANCOVA model assumes that the "true" baseline score is really 
somewhat lower, and that in the absence of any program, participant and comparison group scores would likely 
converge to a degree (regress to the mean). The adjustment for the baseline score in the regression calculation 
takes this assumption into account and adjusts the post-program outcome by less than the full value of the 
baseline score. The difference-in-difference model assumes that there is no regression toward the mean and, by 
subtracting the baseline from the post-program score, gives the baseline score full weight as an adjustment. In 
the case of civic attitudes, where the outcome being measured is likely to have directly affected selection, there 
is a much stronger case to be made that at least some regression toward the mean is likely. If so, by giving the 
baseline score full weight (in effect overadjusting for baseline differences), the difference-in-difference method 
is likely understating the real program effect. In this instance, then, it is useful to view the two methods as 
representing high and low estimates. Again, see the discussion of the two methods in Appendix B for a number 
of specific examples of the differences in the estimates produced by the two methods. 
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most direct measure of student attitudes towards service itself. The questions in that measure focused 
less on general attitudes and more on the degree to which students felt they were aware of needs in 
the community, believed that they could make a difference, knew how to design and implement a 
service project, and were committed to service now and later in life. In this instance, the service 
experience appears to have affected student attitudes most directly, providing a boost in students’ 
understanding of the service task and their confidence in their ability to continue it. 



Impacts on Volunteer Behavior 

The Learn and Serve programs also had a significant positive impact on involvement in 
volunteer service during the program period for all the program participants. Program participants 
were significantly more likely to have been involved in some form of volunteer service and to have 
contributed more hours of service during program participation than students not enrolled in the 
program (see the second panel in Exhibit 3.3). Overall, participants were nearly 20% more likely to 
have been involved in some form of service activity during the previous six months than comparison 
group students, and to have provided more than twice as many hours of service as comparison group 
members during that time period (an average of 73 hours of service versus 32 hours for comparison 
group members). 

The impacts on volunteer behavior were most evident among high school participants, with a 
more limited impact on students in middle school programs. Among high school students, 
participants were 18% more likely to have been involved in service and provided more than three 
times more volunteer hours than comparison group members (78 hours vs. 25 hours). Middle school 
participants were 20% more likely to have been involved in service, but provided only 1.4 times as 
many hours as comparison group members, a difference that was not statistically significant. 

At one level, it is not surprising that participants in a service program should have more 
volunteer hours than non-participants. But it is important to view these findings in context. National 
surveys indicate that 61 % of American teenagers perform at least some volunteer work in the 
community each year, so that both comparison group members and participants are likely to have 
some volunteer experience — indeed, 75% of the comparison group members indicated involvement 
in at least one form of volunteering at baseline. 10 What the findings on volunteer hours and 
participation in this study show is that service-learning programs were not simply diverting students 
from volunteer service that they would otherwise normally be doing. Instead, the data support the 
argument that students in service-learning programs were more likely to be involved in volunteer 
service as a result of program participation, and that they provided significantly more hours of service 
than young people who were not enrolled. 



10 Baseline measures for participants and comparison group members are reported in Appendix B. For 
national figures on volunteering, see The Independent Sector, Volunteering and Giving Among American Teenagers 12 
to 17 Years of Age: Findings from a National Survey , (Washington, D.C.: The Independent Sector, 1992). 
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Impacts on Educational Attitudes and Performance 

The Learn and Serve programs also had positive short-term effects on participants’ educational 
attitudes and school performance, though only on a few of the measures examined in the study. For 
the participant group as a whole, the Learn and Serve programs produced positive, statistically 
significant post-program impacts on two measures — school engagement and math grades — and 
marginally significant impacts (at the .10 level) for science grades and core grade point average 
(English, math, science, social studies grades combined). There were no impacts on English and social 
studies grades, or on measures of course failure, absenteeism, homework hours, or educational 
aspirations. 

As with most of the other measures, high school students showed a slightly broader pattern of 
significant impacts than middle school students, with strongly significant impacts on school 
engagement and math grades and a marginally significant impact on science grades. High school 
students also showed a substantial, statistically significant reduction in course failures, with 
participants 65 % less likely than comparison students to have failed one or more courses during the 
most recent semester. Middle school students showed positive increases in social studies (18%), math 
(12%), and science grades (6%) and core GPA (9%). But the social studies increase was the only 
one that was statistically significant." 

As was the case with the measures of civic attitudes, the ANCOVA analysis does indicate a 
stronger, more widespread pattern of impacts on educational measures, with impacts on math, social 
studies, and science grades, core grade point average, as well as educational aspirations. However, in 
this instance, there is less reason to believe that academic performance figured largely in student 
decisions to select a service-learning program, and the difference-in-difference estimates are likely to 
provide a less biased overall estimate. To the extent that school attitudes and performance did figure 
into the decision to participate, the difference-in-difference estimates can be considered lower bounds. 

The limited nature of the educational impacts makes it difficult to conclude that service-learning 
is having a strong positive impact on students’ educational experience. However, the fact that there 
are impacts on school engagement and small impacts on both individual course grades and core GPA 
for the participant group as a whole does suggest that service-learning is having some degree of 
positive influence on school performance while students are in the program. The most likely 
explanation is that increased student engagement is translating into increased attention and effort at 
school. In that regard, it is interesting to note that the biggest increases are taking place in math 
classes — the subject that students often find least attractive, and the course with the lowest average 
grades at baseline. 

As with civic attitudes, where educational impacts are evident, they are relatively small — a 
10% increase in math grades, a 6.5% increase in science, and a 4% increase in core GPA. The 
change in math grades, for example, represents a increase from a solid C to a C+ (i.e. from a 2.26 



11 The fact that positive impact on grades were evident at high school and not at middle school may reflect the 
smaller middle school sample size. Information on grades was available for 233 middle school students vs. 500- 
600 high school students. 
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average to 2.48). On other hand, given that comparison group students are also in school and 
receiving educational services, even modest impacts can be meaningful. When the various measures 
are taken together, they do suggest that service is helping students to be somewhat more engaged in 
school and, perhaps as a result, do slightly better in their classes. At the same time, it is clear that 
for the participant group as a whole, involvement in service-learning does not promote substantial 
changes in school performance. 12 



Social and Personal Development 

The one area in which there are no statistically significant post-program impacts for participants 
as a whole was on the measures of personal and social development, which include measures of 
perceived communications skills, work orientation, and involvement in risk behaviors. The only 
impact evident for the whole group was a marginally significant impact on teenage pregnancy. There 
are also no significant impacts for high school students. 

For middle school students, however, there is some evidence of impact on involvement in risk 
behaviors, with a marginally significant impact on teenage parenting and a substantial, statistically ~ 
significant impact on arrests for middle school youth. The fact that there are impacts on two risk 
measures for middle school students suggests that for these younger students, involvement in a well- 
organized service-learning program may play a role in reducing some kinds of risk behaviors, though 
not all (there was no impact on reported alcohol consumption, drug use, or delinquent behaviors). 

Similarly, while none of these findings support an argument that service-learning by itself is an 
effective preventative for at-risk behavior, the marginal impact on teenage parenting for middle school 
students and participants as a whole lends some support to the notion that service-learning can 
contribute to a multi-faceted intervention. A growing number of programs for at-risk youth have 
incorporated community service or service-learning into their overall design, and evaluations of 
several of those programs have shown a significant reduction in some risk behaviors, including 
teenage parenting. When coupled with those evaluations, the data from this study suggest that, while 
service alone is not likely to dramatically reduce involvement in risk behaviors, service may 
contribute to the effectiveness of programs targeted to reducing at-risk behaviors among school-aged youth. 13 



12 As will be discussed later in this chapter, for some subgroups of participants the impacts on academic 
performance were larger. Among minority youth, for example (Black, Hispanic and other non-white students), 
the increase in math grades averaged 22%, the increase in science grades was 14%, and the increase in core 
GPA was nearly 10%. All three of these were significantly greater than the increases shown by white students. 
Educationally disadvantaged students saw 15% increases in math and social studies grades, and young women 
saw an increase of 17% in their math grades. While these differences are scattered, they do suggest that while 
students on average may experience a small improvement in school performance through service-learning, some 
students are likely to benefit more substantially. 

13 Two of the programs that have been studied that incorporate service into their program design and have had 
an impact on teenage pregnancy are the Teen Outreach Program (TOPS), a national pregnancy prevention program 
founded by the Junior League, and the Quantum Opportunities Program (QOP), a multi-site dropout prevention 
initiative managed by the Opportunities Industrialization Centers of America and funded by the Ford Foundation. See 
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Impacts on Subgroups 

In addition to the analysis of impacts on participants as a whole and on middle and high school 
students, the evaluation also examined post -program impacts for a number of subgroups in the study, 
including males and females, white and minority students, economically and educationally 
disadvantaged students, students involved in one or more risk behaviors at baseline, and students with 
and without prior volunteer experience or participation in prior service-learning programs. The 
purpose of the analysis was to identify any groups that appeared to particularly benefit (or fail to 
benefit) from service-learning compared to other students. In each instance, we examined the 
outcomes for complementary pairs of subgroups (for example, males and females) to see if there were 
significant differences between the impacts for the two groups, as well as examining the impacts on 
each group individually. 14 

Exhibit 3.4 provides an overview of those measures that showed significant differences in post- 
program impacts between pairs of subgroups (that is, where the impact for one group was 
significantly higher or lower than that of its complementary group). Though there are a number of 
instances where there are differences in impacts between groups, there are few instances where one 
group shows consistently stronger impacts, and in many case, the differences between groups are only 
marginally significant. In general, no one group seems to be consistently more likely to benefit from 
service than another. 

While there are no across-the-board differences in impacts between subgroups, some differences 
do stand out. As noted earlier, minority (non-white) students show significantly stronger impacts on 
measures of academic performance than do white students. These include significantly stronger 
impacts on math and science grades and core grade point average, as well as a marginally stronger 
impact on course failures. Young women also show stronger impacts on several education-related 
measures (Math grades, GPA, and college aspirations), and students who were involved in at-risk 
behavior (alcohol or drug use, etc.) at baseline appear to experience slightly stronger impacts on 
several measures of civic and social attitudes. 

Though these differences suggest that some groups may benefit on some impacts more than 
others, the broader finding is that the benefits of participation in service-learning appear to cut across 
all the various subgroups relatively evenly. Exhibit 3.5 shows the impacts on individual subgroups 
for those measures where there were positive post-program impacts for participants as a whole. As 
that exhibit shows, the impacts of service-learning appear to take place across the board, with most of 
the subpopulations showing positive impacts on most of the measures. Of particular note is the fact 
that participation in service-learning appears to have an impact not only on first-time participants, but 



Joseph Allen, S. Phillaber, and N. Hoggson, "School-Based Prevention of Teenage Pregnancy and School Dropout: 
Process Evaluation of the National Replication of the Teen Outreach Program," American Journal of Community 
Psychology, 18 (1990), 505-524; and Andrew Hahn and Janet Reingold, Quantum Opportunities Program: A Brief on 
the QOP Pilot Program, (Waltham, MA: Brandeis University, Heller Graduate School, September, 1995). 

14 The results presented here are based on the difference-in-difference analysis. The detailed post-program 
results from the subgroup analysis using both the difference-in-difference analysis and an analysis using the ANCOVA 
model are presented in Appendix E. 
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